Papers with systematic logical reasoning
Counterfactual reasoning: Testing language models’ understanding of hypothetical scenarios (2023.acl-short)
Copied to clipboard
| Challenge: | Existing language models have demonstrated remarkable improvements in downstream tasks, but it remains unclear to what extent they are consequences of correlation with linguistic heuristics versus robust reasoning about causal relations grounded on understanding of world knowledge. |
| Approach: | They propose to test language models with counterfactual conditionals to test their ability to distinguish hypothetical scenarios from reality. |
| Outcome: | The proposed model overrides real-world knowledge in counterfactual scenarios, but most models are driven by lexical cues. |